Recognition Confidence Measures: Detection of Misrecognitions and Out-Of-Vocabulary Words

This paper describes and evaluates a new technique for measuring confidence in word strings produced by speech recognition systems. It detects misrecognized and out-of-vocabulary words in spontaneous spoken dialogs. The system uses multiple, diverse knowledge sources including acoustics, semantics, pragmatics and discourse to determine if a word string is misrecognized. When likely misrecognitions are detected, a series of tests distinguishes out-of-vocabulary words from other error sources. The work is part of a larger effort to automatically recognize and understand new words when spoken in a spontaneous spoken dialog. We describe a system that combines newly developed acoustic confidence measures with the semantic, pragmatic and discourse structure knowledge enbodied in the MINDS-II system. The newly developed acoustic confidence metrics output independent probabilites that a word is recognized correctly along with a measure of how reliably we can estimate if a word is wrong. The acoustic confidence metrics are derived from normalized acoustic recognition scores. The acoustic scores are normalized by estimates of the denomiator of Bayes equation. To evaluate the utility of using the acoustic techniques together with higher-level constraints, the preliminary system restricted component interaction. Words with normalized acoustic scores that had a 95% or greater probability of being incorrect were flagged prior to being input to the MINDS-II analysis module. For this study, MINDS-II independently used its higher-level knowledge to detect recognition errors that were semantically or contextually inappropriate. Misrecognized word strings were then re-recognized using an RTN-based speech decoder guided by a dynamically derived, highly constrained grammar that restricts the possible words that can be matched, biasing the recognizer against illogical and highly improbable content. Speaker goals and plans, contextual appropriateness, discourse and spontaneous speech structure are all considered in the derivation of grammars. A grammar is dynamically derived for each string of misrecognized words encountered within an utterance and essentially defines a set of semantic content predictions for the word string. Although a rudimentary procedure was employed to estimate the conjoined effect of merging these two knowledge sources, the results indicate that the conjoined usage of normalized acoustic confidence measures of accuracy and the higherlevel, semantic, pragmatic and discourse level constraints embodied in the MINDS-II system enables the larger system to overcomes the weaknesses of each individual technique. The techniques detect complementary phenomena. Significantly more recognition errors and out-of-vocabulary words are detected when both of these techniques are used together than when either is used alone. Alone, the acoustic methods can only detect between 2/3 and 3/4 of the recognition errors. Similarly, the higher-level constraint based methods cannot detect contextually consistent misrecogntions. Together, however, the acoustic methods detect the significant misrecognized content words that are missed by the higher-level, knowledge-based techniques. Further, the knowledge-based techniques detect most of the mid-utterance corrections and interjections as well as many of the confusible and small words that are missed by the acoustic methods. Current work focuses upon development of more sophisticated techniques for conjoining these two knowledge sources.